On Selecting the Best Unsupervised Evaluation Techniques for Image Segmentation

One fundamental difficulty with evaluation of segmentation is that there is no objective, clear definition of good or bad segmentation. Even worse, different observers often do not agree on how to segment the same image. In this paper, we present six unsupervised metrics in the literature that are commonly used to evaluate segmentation results. Then we propose a framework to find the best comparison metric in the sense that this metric is the most consistent with the ground-truth provided by manual segmentation and, at the same time, is the most sensitive to random segmentation results. We believe a “good” metric should produce a high score on the ground-truth segmentation, as well as produce a low score on random segmentation. We employ the best unsupervised metric to compare results from different image segmentation methods on the Berkeley Segmentation Dataset and Benchmark, which consists of 300 color images of natural scenes. Keywords— image segmentation, unsupervised evaluation, quantitative evaluation